Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of TRT wrapping via inference.json #620

Merged
merged 38 commits into from
Sep 10, 2024
Merged

Implementation of TRT wrapping via inference.json #620

merged 38 commits into from
Sep 10, 2024

Conversation

borisfom
Copy link
Contributor

Sample implementation of using new trt_wrap() from MONAI
Depends on Project-MONAI/MONAI#7990

@borisfom borisfom mentioned this pull request Aug 16, 2024
Open
@borisfom
Copy link
Contributor Author

I observe 2.5x speedup with FP16 TRT for a small sample (28.57it/s vs 11it/s). Did not check the accuracy.

borisfom and others added 12 commits August 22, 2024 22:34
@borisfom
Renamed trt_wrap -> trt_compile
f7da0ec 
Signed-off-by: Boris Fomitchev <[email protected]>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
c8d64bf 
…wrapper

Signed-off-by: Boris Fomitchev <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
aab37c5 
for more information, see https://pre-commit.ci
@borisfom
Added TRT for pathology_nuclei_segmentation_classification
7b64083 
Signed-off-by: Boris Fomitchev <[email protected]>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
43da1f5 
…wrapper
@borisfom
Merge remote-tracking branch 'origin/dev' into trt_wrapper
3d8a738
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
4a9f2ba 
for more information, see https://pre-commit.ci
@borisfom
Merge remote-tracking branch 'origin/dev' into trt_wrapper
7496dfb
@borisfom
Added Vista2D TRT config
337fcf3 
Signed-off-by: Boris Fomitchev <[email protected]>
@borisfom
Cleaned up using new config syntax, batch size added
16f720f 
Signed-off-by: Boris Fomitchev <[email protected]>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
6cc423d 
…wrapper

Signed-off-by: Boris Fomitchev <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
a205bf4 
for more information, see https://pre-commit.ci
@borisfom borisfom changed the title POC implementation of TRT wrapping via inference.json Implementation of TRT wrapping via inference.json Aug 27, 2024
@borisfom
Copy link
Contributor Author

borisfom commented Sep 6, 2024

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default
python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

@borisfom borisfom mentioned this pull request Sep 6, 2024
Closed
8 tasks
@binliunls
Copy link
Contributor

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

Hi @borisfom ,
Thank you for your update. As shown in the inference config file, you should compile the diffusion and autoencoder with TRT instead of network, since the network parameter is not defined in either bundle. Please check if you can compile both diffusion and autoencoder.

Thanks,
Bin

@borisfom
Copy link
Contributor Author

borisfom commented Sep 6, 2024

@binliunls : Thanks, you are correct! Works as expected now.

@yiheng-wang-nv
update metadata
f0a9329 
Signed-off-by: Yiheng Wang <[email protected]>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
update mgpu test script
bde5d52 
Signed-off-by: Yiheng Wang <[email protected]>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
use nvidia host large files
3e80c93 
Signed-off-by: Yiheng Wang <[email protected]>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
update vista2d
35e697c 
Signed-off-by: Yiheng Wang <[email protected]>
@yiheng-wang-nv yiheng-wang-nv mentioned this pull request Sep 10, 2024
Closed
@yiheng-wang-nv
Copy link
Collaborator

/build

1 similar comment
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv yiheng-wang-nv enabled auto-merge (squash) September 10, 2024 06:28
@yiheng-wang-nv yiheng-wang-nv merged commit efadc57 into Project-MONAI:dev Sep 10, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KumoLiu KumoLiu KumoLiu left review comments

@yiheng-wang-nv yiheng-wang-nv yiheng-wang-nv approved these changes

@Nic-Ma Nic-Ma Nic-Ma approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@borisfom @KumoLiu @yiheng-wang-nv @Nic-Ma @binliunls